Establishing Performance Baselines for Text Understanding Systems
نویسنده
چکیده
A task-oriented evaluation of text understanding systems was prepared and conducted. Nine different NLP systems participated in the evaluation. NOSC collected 150 texts to be used as development (i.e. training) and test data and prepared explanatory documentation on them. The performance task--a simulated database update task--and the expected outputs for each text were defined. A scoring system was devised and underwent considerable revision in the course of the evaluation.
منابع مشابه
Identifying Condition-Action Statements in Medical Guidelines Using Domain-Independent Features
This paper advances the state of the art in text understanding of medical guidelines by releasing two new annotated clinical guidelines datasets, and establishing baselines for using machine learning to extract condition-action pairs. In contrast to prior work that relies on manually created rules, we report experiment with several supervised machine learning techniques to classify sentences as...
متن کاملMTI for Full Text
To provide a stable base for the experiments with full text, the MTI indexing paths were run separately on each of the sections of the full text test collection. The output from each indexing path was saved and subsequently used by MTI for all of the experimental processing. The evaluation in the phase 1 experiments reported in the AMIA paper was based on the human indexing extracted from MEDLI...
متن کاملLanguage Understanding for Text-based Games using Deep Reinforcement Learning
In this paper, we consider the task of learning control policies for text-based games. In these games, all interactions in the virtual world are through text and the underlying state is not observed. The resulting language barrier makes such environments challenging for automatic game players. We employ a deep reinforcement learning framework to jointly learn state representations and action po...
متن کاملDESIGN AND IMPLEMENTATION OF FUZZY EXPERT SYSTEM FOR REAL ESTATE RECOMMENDATION
<span style="color: #000000; font-family: Tahoma, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: justify; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none; backgro...
متن کاملActive baselining in passive data environments
In order to decide if systems are running according to their usual trend, it is necessary to compare against a performance baseline that defines an operating envelope. We describe how baselines can be derived from passive stores of performance data, which are typically flat text files or databases. Baselines can be built as needed by varying the baseline norm, granularity, update frequency, etc...
متن کامل